DOCUMENT RESUME 



ED 351 178 



RC 018 860 



AUTHOR 
TITLE 



INSTITUTION 



REPORT NO 
PUB DATE 
NOTE 



AVAILABLE FROM 



PUB TYPE 



Huang, Zheng Sen 

Report on Student Performance in the Northern 
Territory Primary Assessment Program for Aboriginal 
Schools: 1987. ' ~ ~ " " 

Northern Territory Dept. of Education, Darwin 
(Austral ia) . 
"lSBN-0-72A5~ 1965-3 
Apr 89 

23p.; Report presented at the Meeting of the Rural 
Education Research A.ssociation (Alice Springs, 
Northern Territory, Australia, February 1992). For a 
related report, see RC 018 863. 

Curriculum L Assessment Branch, GPO Box A821 Darwin, 
Northern Territory 0801, Australia (Stock number 
810-0017). 

Reports - Evaluative/Feasibility (1A2) — 
Speeches/Conference Papers (150) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

Academic Achievement; Achievement Tests; Educational 
Testing; Foreign Countries; ^Indigenous Populations; 
Intermediate Grades; ^Mathematics Achievement; 
Program Evaluation; '^^Reading Achievement; Rural 
Education; ^Rural Schools; Secondary Education; 
Student Evaluation; Test Results; *Test Score 
Decline 

Aboriginal People; ^Aboriginal Schools (Australia); 
^Australia (Northern Territory) 



ABSTRACT 

This report gives the results of the Primary 
Assessment Program (PAP) in Aboriginal schools (Australia) for the 
1987 school year. The purpose of PAP is to provide instructional 
feedback on student achievement in reading and mathematics and to 
provide indicators of performance for public reporting. All 
Aboriginal schools which had students in grade 5 to post-primary were 
asked to administer five reading tests and eight mathematics tests. 
Results indicate the following: (1) participation rate was low as 
less than a quarter of the target population of 2,500 students took 
the tests durirg 1987; (2) many schools either excluded all, or a 
substantial number of their students from testing as The Teachers 
Federation banned the testing program; (3) performance in practically 
all areas tested was considered low; and (A) when the test results 
for 1987 were compared to 1986, there was a significant drop in 
performance in 6 of 17 objectives tested in mathematics and a 
significant drop in performance in 6 of 10 reading tests. These 
results suggest that remedial strategies need to be developed and 
implemented at both the school and system level. This report also 
su^^f^sts that research into the causes for poor student performance 
be initiated. This report contains numerous data tables. (LP) 
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SECTION 1 
TEST ADMINISTRATION 



1,1 INTRODUCTION 

The program which was introduced in 1986 was designed to provide schools 
with quality assessment materials for assessing achievement in the core 
objectives in English and mathematics. Reading and mathematics 
achievements are measured using paper and pencil tests; writing is 
assessed using a moderation process. 

Por system wide testing and public reporting of results, the program 
requires schools to administer a selection of tests in reading and 
mathematics and return students' answer sheets. Schools are issued with 
a printout of the results of their own students and the Territory, 
results for comparison. This report gives the results of performance on 
the reading and mathematics tests conducted towards the end of 1987. 



1.2 PURPOSE 

The purpose of the program is two-fold: 

(i) to provide instructional feedback on achievement in the core 
objectives in English and mathematics; 

Mi) to provide at the system level some indicators of performance for 
public reporting. 



1,3 COLLECTION OF DATA 

All Aboriginal schools which had students in Years 5 to Post Primary 
were included. Using a stratified random sampling procedure, schools 
were assigned into two groups, each group broadly representative of the 
total population of Aboriginal schools. 

Each school was asked to administer five reading tests and eight 
mathematics tests over a period of time during the second semester. 
Students from Years 5 to Post Primary working at or beyond Stage 5 of 
the English core objectives and Phase 2 of the mathematics core 
objectives were tested. Those operating much below these levels were 
excluded from the testing. 



1.4 ADMINISTRATION OF THE TESTS 

A set of guidelines was pr'ovided to teachers to ensure some uniformity 
in test administration. Students were allowed 2 to 3 minutes to look at 
the test. The teachers explained how students should write their 
answers for multiple-choice and open-ended items. Students were not 
allowed to talk with each other during the test and teachers were not to 
help students answer any test questions. 
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However, no time limits were specified: teachers were asked to use 
their own judgement in deciding whether enough time had been given. The 
timing and sequence of testing were also to be decided by the classroom 
teacher. . 



1,5 CONFIDENTIALITY OF INFORMATION 

To protect the confidentiality of information collected, each school was 
given a school code known only to the project co-ordinator. Each 
student was also given a student number by the school. The student used 
the same number for all the tests that he or she took. The school code 
and student number were used to identify each answer sheet keyed into 
the computer for data processing and analysis. 
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SECTION 2 

ANALYSIS OF PERFORMANCE ON THE MATHEMATICS TESTS 



2.1 TABULATION OF RESULTS 

Two sets of tables have been presented. Table lA shows proportions 
(percentages) of students achieving at various cutoff scores, namely, 
70%, 80%, 90% and 100% correct. The cutoff score for mastery of an 
objective was 80% correct. 

Table IB shows changes in performance between 1986 and 1987 for each 
objective tested. Changes in performance that were statistically 
significant are indicated with an asterisk (*) beside them- 

2.2 PERFORMANCE BY CONTENT AREAS 

Using the 80% cutoff score as the level for success, the mean percent 
successful for each strand was calculated. The strands covered by the 
tests were space, measurement and number. 

(a) Space 

The tests included in the testing program measured students' 
ability to identify 2-dimensional figures and use ordered pairs to 
locate positions in space. Performance in this area was markedly 
higher than those in the measurement and number strands. 

Mean percent successful = 53.8 

(b) Measurement 

Five tests irr measurement were included which required students to 
recall facts, identify symbols for units of length and mass, 
recall units used for length and mass, read and write time shown 
on a clockface. 

The measurement tests proved to be very difficult for most 
students. Results showed that in each of the tests, a very high 
proportion was unsuccessful. 

Mean percent successful = 19.4 

(c) Number 

There were eight tests on simple, routine computations, one test 
on place values and one on reading bar graphs. 

The success rate was low on practically all areas. 

Mean percent successful = 34.8 
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2.3 SUMMARY OF RESULTS BY STRANDS 

The following table provides a summary of the results in relation to the 
- content areas-. — - - : - - ^- - - — 



STRAND NUMBER OF OBJECTIVES MEAN % SUCCESSFUL 



Space 2 53.8 

Measurement 5 19.4 

Number 10 34.8 



2.4 CHANGES IN PERFORMANCE BETWEEN 1986 AND 1987 

The results from the 1986 and 1987 administrations were compared based 
on the tests common to both years. As Table IB shows, most of the gains 
or losses between the two years were only marginal and not statistically 
significant. From Table IB, the changes in performance between 1986 and 
1987 may be summarised as follows: 

CHANGES IN PERFORMANCE 

SIGNIFICANT SIGNIFICANT NO 

IMPROVEMENT DECLINE CHANGE 

Number of Objectives 1 6 10 

2.5 SIGNIFICANT CHANGES 

The results indicated a significant improvement in 1987 in one of the 
areas in the space strand i.e. identify 2-D shapes. 

However, there appeared to be a significant drop in performance in 
respect of six objectives, one in the space strand and the other five in 
the number strand. 

In 1987, the results in respect of the following objectives appeared to 
be significantly worse: 

(i) Use ordered pairs to locate position in space. 

(ii) Subtract one digit number from a number less than 100. 

(iii) Add, up to 3 digit numbers, with one or two decimal places. 

(iv) Multiply a two-digit whole number by a single digit number, with 
or without regrouping. 

(v) Divide a two-digit number by a one-digit number without 
regrouping, no remainder. 

(vi) Identify units, tens and hundreds in numerals up to 1000. 
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TABLE lA: PERCENTAGES AT DIFFERENT CUTOFFS: MATHEMATICS 
YEAR 5 TO POST PRIMARY (1987) 



No Test Code Qb.i ectives No of Proportions Obtaining 

Items 70% 80% 90% 100% Correct 

1. S-A-1 Identify 2-0 shapes (polygons) 10 74.9 65,8 52.5 27.4 

2. S-A-2A Use ordered pairs to locate 

position in space 10 55.0 41.8 36.6 25.5 



MEASUREMENT 

M-0-1 Recall some facts, e.g. 

100 cm * 1 m 
1 000 g « 1kg 
24 h = 1 day 

60 min - 1 hour 10 27.4 J^i 9.6 . 5.0 

M-0-2 Identify the following 

symbols: m. cm, L, g and kg 5 23.2 li^ 10.1 10.1 

M-0-3 Given an everyday object, 

identify the unit for its 
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measure: m, cm, L, g or kg 


5 


17. 


9 


17-9 


9. 


6 


9.6 


6. 


M-C-7A 


Read time on a clockface. 


5 


15 


0 




5 


0 


5.0 


7. 


M-C-7B 


Write time on a clockface 


5 


21 


1 


ZlA 


7 


5 


7.5 






NUMBER 
















8. 


N-A-3A 


Subtract one digit number from 




















a number less than 100 


10 


42 


9 


28.1 


21 


.9 


9.5 


9. 


N-B-1 


Add, up to 3 digit numbers, 




















with regrouping 


10 


56 


.8 


5Q.? 


44 


.1 


27.3 



10. N-B-2 Subtract numbers, up to 

3 digits, regrouping only in 

the units 10 30.0 21^ 18.6 13.6 

11. N-B-3 Add, up to 3 digit numbers, 

with one or two decimal 

places 10 39.2 ilLQ 13-3 8.3 
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TABLE lA: PERCENTAGES AT DIFFERENT CUTOFFS: MATHEMATICS 
YEAR 5 TO POST PRIMARY (1987) 



Test Code Qb.lectives 



12. N-8-4 



13. N-C-IB 



14. 



N-0-2 



15. N-0-3A 



16. N-E-2B 



17. N-F-2 



No of 



MMS£R 



Subtract 3 digit numbers 
with one or two decimal places, 
regrouping in the first column 
only 10 



Recall basic facts in division, 
quotients to 10, single digit 
divisor and no remainder 

Multiply two-digit whole 
digit numbers by single digit 
numbers, with or without 
regrouping 

Divide a two-digit number by 
a one-digit number without 
regrouping, no remainder 

Identify units, tens and 
hundreds in numerals up to 
1 000 

Read a bar graph 



10 



10 

10 

5 
5 



Proportions Obtaining 

m m m iqo^ correct 



29.3 ZLJ 12.7 8.5 



56.1 47.8 32.0 



21.5 iLi 13.4 8.9 

33.7 ILJ 20.5 10.8 

55.3 iU 19.7 19.7 

39.0 iU) 21.5 21.5 
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TABLE IB: CHANGES IN PERFORMANCE 
MATHEMATICS-, YEAR 5 TO POST PRIMARY 



No Test Code Obi actives No of Percentaoe Achieving Competence Change 

Items N 198fif%^ H 1987(%) 1986-87 

iPA££ 





S-A-1 


Identify 2-0 shapes (polygons) 


10 


226 


57.5 


219 


65. 


8 


+8.3* 


2. 


S-A-2A 


Use ordered pairs to locate 
position in space 

^ MEASUREMENT 


10 


320 


70.6 


273 


41. 


8 


-28.8* 


3. 

* 


M-0-1 


Recall some facts, e.g. 
100 cm = 1 m 
1 000 g « 1kg 
24 h » 1 day 
60 min =■ 1 hour 


10 


209 


19.1 


219 


19 


6 


+0.5 


4. 

ft 


M-O-2 


Identify the following 
symbols : m, cm, L, g and kg 


5 


293 


24.2 


267 


23 


.2 


-1.0 


5. 


M-0-3 


Given an everyday object, 
identify the unit for its 
measure; m, cm, L, g or kg 


5 


208 


18.3 


240 


17 


.9 


-0.4 


6. 


M-C-7A 


Read time on a clockface. 


5 


284 


13.4 


220 


15 


.0 


+1.6 


7. 


M-C-78 


Write time on a clockface 


5 


307 


16.0 


213 


21 


.1 


+5.1 



NUMBER 

8. N-A-3A Subtract one digit number from 

a number less than 100 10 234 48.7 210 28.1 -20.6* 

9. N-8-1 Add, up to 3 digit numbers, 

with regrouping 10 234 45.3 227 50.2 -4.9 

10. N-B-2 Subtract numbers, up to 

3 digits, regrouping only in 

the units 10 373 32.2 220 25.9 -6.3 

11. N-8-3 Add, up to 3 digit numbers, 

with one or two decimal 

■placss ■ 10 ■ • 281 • 52. 7 240 30.0 -22.7' 
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TABLE IB: CHANGES IN PERFORMANCE 
MATHEMATICS, YEAR 5 TO POST PRIMARY 



HQ 



12. 



13. 



Test Code Ob.iectlves 
NUMBER 



N-C-IB 



14. H-O-Z 



15, N-0-3A 



16, N-E-28 



Items 



Subtract 3 digit numbers 
with one or two decimal places, 
regrouping in the first column 
only 10 



17, 



N-F-2 



Recall basic facts in division, 
quotients to 10, single digit 
divisor and no remainder 

Multiply two-digit whole 
digit numbers by single digit 
numbers, with or without 
regrouping 

Divide a two-digit number by 
a one-digit number without 
regrouping, no remainder 

Identify units, tens and 
hundreds in numerals up to 
I 000 

Read a bar graph 



10 



10 

10 

5 
5 



Percentage Achieving Competence Change 
1 - 1986f%) - N 1987 f%) 1986-87 - 



203 26,6 212 21.7 



281 59.4 278 52.9 



299 31.4 247 17.4 



223 39.9 166 27,1 



-4.9 



-6.5 



•14. 0'^ 



-12.8* 



291 73.5 300 55.3 -18,2* 
193 43.5 223 39.0 -4.5 



Note: 

M » Number in sample 

* « Statistically significant 
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SECTION 3 

ANALYSIS OF PFRFORMANCE ON THE READ TNfi TESTS 



3.1 TABULATION OF RESULTS 

Two sets of tables have been presented. Table 2A shows proportions . 

(percentages) obtaining various cutoff scores, viz. 60%, 70% 80% and 
100% correct. The cutoff score for mastery was 70%. 

Table 2B provides data on percentages achieving competence on each test 
for 1985 and 1987 respectively and changes in performance between the 
two years. Changes that were statistically significant are shown with 
an asterisk (*) beside them. 

3-2 DISCUSSION OF THE RESULTS 

Ten separate tests were administered using a sampling matrix in which 
• each school had to administer two tests of reading comprehension, one 
test on dictionary skills and two tests of basic literacy. 

Mean percent 'successful' (where success was 70% correct) was calculated 
for each of the three categories and reported below. 



CATEGORY 

(a) Prose comprehension 
(C series) 

(b) Dictionary Ski 1 1 s 
(D series) 

(c) Basic Literacy 
(R series) 



The results seem to suggest that most students had difficulty in all 
categories: comprehension of prose passages, dictionary (vocabulary) 
and basic literacy. Success rates were very low. 



MEAN % SUCCESSFUL 
23.0 



33.9 
38.7 
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3.3 CHANGES IN PERFORMANCE BETWEEN 1986 AND 1987 • 

The changes in perfonnance between 1986 and '1987 are depicted in Table 
2B. Results for 1987 appeared to be somewhat worse than those of the 
previous year as in six of ten tests administered, the drop in 
performance" was statisticaTly" significant. Changes in" performance on" 
the remaining four tests were marginal and did not appear to be 
statistically significant. 

The changes in performance between the two years may be summarised as 
fol lows : 



CHANGES IN PERFORMANCE 



CATEGORY 



SIGNIFICANT 
IMPROVEMENT 



SIGNIFICANT 
DECLINE 



NO 

CHANGE 



(a) 



Prose comprehension 
(C series) 



3. 



1 



(b) 



Dictionary Skills 
(D series) 



2 



(c) 



Basic Literacy 
(R series) 



1 



3 
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TABLE 2A: PROPORTIONS AT DIFFERENT CUTOFFS : READING 
YEAR FIVE TO POST PRIMARY (198?) 



ilS Test Code Jyp^? 



No of Proportions Obtaining 

Items 60% 70% 80% 100% Correct 



1. C2 

2. C8 

3. C9 

4. Cll 



Formal 

Prose Comprehension 
Passages 



13 
10 
12 
12 



33.1 iZJ) 
37.3 
28.8 
33.6 



11.2 
20.1 
13.3 
19.6 



1.6 

6.2 
2.2 
3.3 



5. 01 

6. 03 



Dictionary Skills 



5 74.4 58.8 38.4 

9 16.4 2JI '^■'^ 2.7 



R7 
R9 
RU 



10. R16 



RDP Magazine Contents 
RDP Addressed Envelope 
RDP Street Map 
RDP Excursion Timetable 



8 
10 
10 
10 



42.6 ISLi 
63.9 5U 

51.7 2SL3 
51.5 2Ld 



10.5 3.4 
40.8 12.7 



24.9 
24,2 



7.2 
1.5 
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TABLE 2B: CHANGES IN PERFORMANCE 
REA.OING, YEAR FIVE TO POST PRIMARY 



as 



Test Code Types 



No of Percentage Achieving Competence " Change 
lim N 1986f%) [j 1987 1986-87 



1. 
2. 
3. 
4. 



C2 

C8 
C9 
Cll 



Formal 

Prose Comprehension 
Passages 



13 

10 
12 
12 



275 ■ 
227 
275 
243 



24.7 
41.4 
34.6 
32.9 



323 
209 
323 
214 



17.0 
26.3 
18.6 
29.9 



-7.7* 
-15. 1* 
-16.0* 

-3.0 



5. 01 



03 



Dictionary Skills 



218 69.3 
298 23.2 



211 58.8 
299 9.0 



•10.5* 
■14.2^ 



7. 
8. 
9. 
10. 



R7 
R9 
RU 
R16 



ROP Magazine Contents 
RDP Addressed Envelope 
RDP Street Map 
ROP Excursion Timetable 



8 
10 
10 
10 



216 
293 
232 
295 



42.1 
56.3 
33.2 
43.1 



209 
338 
209 
326 



25.4 
53.3 
38.8 
37.1 



-16.7* 
-3.0 
+5.6 
-6.0 



Note : 

N - Number in sample 

* ' Statistically significant 
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SECTION 4 
SUMMARY AND CONCLUSION 



4.1 Less than a quarter of the target population of around 2500 students - 
took the tests. Only a third of the total number of schools sent in 
their students' answer sheets for data analysis. The low participation 
rate of the last two years, 1986 and 1987, continued to be a disturbing 
factor. 



4.2 Many schools either excluded all their students or a substantial number 
because in the teachers' opinion, their children were not operating at 
the level of the core objectives. In their dispute with the Department 
over teaching conditions, the Teachers Federation imposed a ban on the 
testing program. The effects of the ban on the program were not exactly 
known . 

4*3 The tests were based on the core objectives for mathematics and reading 
for Year 5, and the benchmarks for success were 80% correct for the 
mathematics tests and 70% correct for reading. Results were reported in 
terms of percentages obtaining scores of 60%, 70%, 80% or 100% correct 
on each test. Schools were issued with the results of their own 
students in April 1988. 



4.4 Performance on the mathematics tests indicated that an average of 54% 
were successful on the space tests, 19% were successful on the 
measurement tests and 35% passed the number tests. A higli proportion of 
students could not perform successfully most of the tasks in the 
measurement area, such as recall of facts : 1000 g = 1 kg or 60 minutes 
= 1 hour, or read and write time correctly. Most students failed the 
addition, subtraction, multiplication and division tasks involving 
basic, routine calculations. There were no word problems- 

When the results for 1987 were compared with those for 1986, there was a 
significant drop in perforinance in six of seventeen objectives tested in 
mathematics . 



4.5 The results in the reading tests were also poor. An average of 23% were 
successful in the reading comprehension tests based on selected prose 
passages. An average of 34% passed the dictionary tests and 39% 
succeeded on tasks related to basic literacy. When results for 1987 and 
1986 were compared, there was a significant drop in performance in six 
of ten reading tests administered in both years. 



4.5 The majority of children had not mastered the main core objectives in 
reading comprehension such as identifying the main idea of a story or 
paragraph, arranging events in a sequence, giving the meaning of words 
in context or inferring from facts stated. The passages used were 
considered to be of appropriate length and difficulty for children in 
Year 5. 
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4.7 The tests in reading for different purposes measured skills related to 
functional literacy quite often demanded in everyday situations for 
performing various tasks. The tasks selected in this particular case 
were extracting information from the contents page of a magazine 
produced for Aboriginal children, reading a hypothetical street map, 
extracting information from an excursion timetable and making sense of 
what was written on the outside of an addressed envelope. 



4.8 There were concerns that these tests were Western-oriented and thus 
culture-biased. It may be argned that there might be factors in the 
Aboriginal culture and learning styles which could affect the validity 
of these tests and hence reduce their reliability and usefulness as 
assessment instruments. However, no achievement test that has been 
produced commercially is absolutely culture-free. It is perhaps true 
that cultures which are more test-oriented seem to possess an advantage. 
The tests were written by people who had taught in Aboriginal schools 
and a lot of care was taken to strip off elements in the tests that 
appeared to be culture-biased. For instance, the passages for 
comprehension came from Aboriginal folklore and the materials testing 
basic literacy were written for Aboriginal children. 

Although the results should be regarded as tentative, it cannot be 
denied that performance in practically all areas was low. Furthermore, 
as the benchmarks were set at the level of the. core objectives for Year 
5, the results should give cause for concern. The data collected in the 
last two years may provide a basis for remedial strategies to be 
developed at both the school and system levels. Research into the 
causes for poor performance seems vital. 
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